I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Related discussion: #16092

To ensure smoother user experience when cluster resource is insufficient, this PR introduces cluster limit in kernel. At the 1st stage, the cluster limit only contains limit on actor count per worker parallelism and there are a lot of headroom for the default limit. The behavior is:

• soft limit is configured via meta config (default to 100)
• hard limit is configured via meta config (default to 400)
• limit can be bypassed via session config

When the soft limit is hit, CREATE MV/TABLE/SINK can succeed with a notice message attached.
When the hard limit is hit, CREATE MV/TABLE/SINK cannot succeed with a notice message attached.

Example:

dev=> create materialized view test_cluster_limit as select * from test;
CREATE_MATERIALIZED_VIEW

dev=> create materialized view test_cluster_limit_exceed_soft as select * from test;
NOTICE:  
- Actor count per parallelism exceeds the recommended limit.
- Depending on your workload, this may overload the cluster and cause performance/stability issues. Scaling the cluster is recommended.
- Contact us via slack or https://risingwave.com/contact-us/ for further enquiry.
- You can bypass this check via SQL `SET bypass_cluster_limits TO true`.
- You can check actor count distribution via SQL `SELECT * FROM rw_worker_actor_count`.
ActorCountPerParallelism { critical limit: 8, recommended limit: 7. worker_id_to_actor_count: ["1 -> WorkerActorCount { actor_count: 32, parallelism: 4 }"] }
CREATE_MATERIALIZED_VIEW

dev=> create materialized view test_cluster_limit_exceed_hard as select * from test;
ERROR:  Failed to run the query

Caused by:
  Protocol error: 
- Actor count per parallelism exceeds the critical limit.
- Depending on your workload, this may overload the cluster and cause performance/stability issues. Please scale the cluster before proceeding!
- Contact us via slack or https://risingwave.com/contact-us/ for further enquiry.
- You can bypass this check via SQL `SET bypass_cluster_limits TO true`.
- You can check actor count distribution via SQL `SELECT * FROM rw_worker_actor_count`.
ActorCountPerParallelism { critical limit: 7, recommended limit: 6. worker_id_to_actor_count: ["1 -> WorkerActorCount { actor_count: 32, parallelism: 4 }"] }

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

A limit on actor count per parallelism (i.e. CPU core) is added to warn user on CREATE MATERIALIZED VIEW/TABLE/SINK statements to ensure smoother user experience when cluster resource is insufficient. Both a soft and hard limit are introduced:

• meta_actor_cnt_per_worker_parallelism_soft_limit (default to 100): when the limit is exceeded, CREATE MATERIALIZED VIEW/TABLE/SINK can still succeed but a SQL notice message will be included when the statement returns.
• meta_actor_cnt_per_worker_parallelism_hard_limit (default to 400): when the limit is exceeded, CREATE MATERIALIZED VIEW/TABLE/SINK will fail and a error message will be included when the statement returns.

Notes:

• The limit will not affect existing created database object in the cluster. In other words, there will be no disruption on MVs/TABLEs/SINKs that are already created. It will only affect future CREATE MATERIALIZED VIEW/TABLE/SINK.
• Scaling the cluster or reducing the streaming job parallelism is the preferred way to resolve the limits.
• The default limits are subjective to change in future releases.
• If users believe it is safe for the cluster to run with the limits exceeded, there are two ways to override the behavior
  • Bypass the check via session variable: SET bypass_cluster_limits TO true.
  • Increase the limit via meta developer config:

  [meta.developer]
  meta_actor_cnt_per_worker_parallelism_soft_limit = 100
  meta_actor_cnt_per_worker_parallelism_hard_limit = 400

  However, keep in mind the cluster can be overload with potential degradation in stability/availability/performance if the check is bypass or the limits are manually increased. Please operate at your own risk with cautions.